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Division  of  Applied  Mathematics 
Brown  University 
Providence,  RI  02912 


Abstract 

This  is  an  AASERT  grant  associated  with  AFOSR  grant  F49620-93- 1-0090,  with  principal 
investigator  David  Gottlieb.  This  grant  has  supported  one  graduate  student  over  three 
years.  The  research  topic  is  parallel  spectral  methods  for  complex  geometries.  The  Ph.D. 
thesis  of  Gerald  W.  Kruse  was  prepared  under  this  grant.  In  the  last  year,  after  the 
graduation  of  Kruse,  the  work  was  continued  by  another  graduate  student,  Henry  Tufo. 
One  conference  proceeding  and  one  journal  article  resulted  from  this  work. 


Statement  of  Problem  Studied 

The  problems  studied  in  this  project  involved  the  development  of  spectral  element  methods 
for  simulation  of  incompressible  flows  in  three-dimensional  geometries.  This  included  the 
development  of  non- conforming  spectral  elements  which  allow  for  flexible  mesh  generation 
and  non-propagating  refinement  (important  for  adaptivity),  the  development  of  fast  domain 
decomposition  solvers  for  large  scale  parallel  applications,  and  physical  applications. 

An  important  component  of  the  work  of  Tufo  was  a  parallel  coarse  grid  solver  which  is 
currently  one  of  the  fastest  available. 


Summary  of  Research  Results 

The  spectral  element  method  is  a  high-order  weighted  residual  technique  which  is  essentially 
a  cross  between  classical  spectral  and  finite  element  methods.  The  computational  domain  of 
interest  is  decomposed  into  relatively  large  parallelepipeds  (bricks)  and  variables  within  each 
subdomain  are  expressed  in  terms  of  high-order  tensor-product  polynomial  bases.  The  use 
of  tensor-product  structures  is  crucial  to  the  vector  performance  of  the  algorithm.  Moreover, 


the  local  structure  of  the  high-order  brick  elements  reduces  memory  latency  since  pointers 
are  not  required  for  each  gridpoint  in  the  domain. 

To  match  complex  domain  configurations  the  spectral  elements  may  be  slightly  de¬ 
formed,  provided  there  exists  an  invertible  mapping  to  a  cubicle  reference  domain.  While 
deformation  provides  a  significant  increase  in  the  meshing  capabilities,  it  is  still  somewhat 
limited  when  isolated  patches  of  mesh  refinement  are  called  for  during  the  course  of  a  simula¬ 
tion.  To  augment  this,  a  nonconforming  spectral  element  method  has  been  developed  which 
allows  for  several  elements  to  join  to  the  edge  or  face  of  another  spectral  element.  Thus, 
there  can  be  a  rapid  transition  in  spatial  resolution  without  additional  mesh  deformation. 

Since  the  parallel  implementation  of  the  spectral  element  method  is  based  upon  a  domain 
decomposition  strategy  in  which  groups  of  elements  are  distributed  to  individual  processors, 
the  nonconforming  extension  to  multiple-element  interfaces  has  a  non-trivial  impact  on 
parallel  communication.  To  simplify  the  communication  software,  a  general  purpose  gather- 
scatter  routine  was  written  which  could  handle  arbitrary  mesh  connectivity  through  a  single, 
very  simple,  user  interface.  This  gather-scatter  software  was  originally  written  in  Fortran, 
but  has  since  been  re-written  by  graduate  student  H.  Tufo  in  C  as  a  very  robust  general 
purpose  package  which  is  currently  being  used  by  several  software  development  groups. 

An  additional  issue  resolved  in  the  thesis  of  Kruse  was  the  development  of  a  diagonal 
mass  matrix  for  the  nonconforming  spectral  element  methods.  This  is  essential  for  time- 
dependent  problems  as  inversion  of  the  mass-matrix  is  required  repeatedly.  For  his  thesis 
work,  Kruse  simulated  the  problem  of  a  high-Reynolds  number  vortex-pair  approaching 
the  ground,  which  is  an  external  flow  problem  requiring  a  highly-varying  mesh,  and  the 
formation  of  horse-shoe  vortices  at  the  base  of  an  end-mounted  cylinder. 

The  thesis  of  H.  Tufo  focussed  primarily  on  fast  parallel  solution  algorithms  for  elliptic 
problems  as  this  is  a  gating  issue  for  incompressible  flow  simulations  in  three-dimensions. 
To  improve  the  iterative  performance  of  the  elliptic  pressure  solve,  a  domain-decomposition 
based  preconditioner  was  developed  based  upon  the  additive  Schwarz  procedure  of  Dryja 
and  Widlund.  One  computes  the  solution  to  the  original  problem  on  independent  over¬ 
lapping  subdomains  (natural  extensions  of  the  spectral  elements)  and  adds  these  solutions 
together.  In  addition,  a  global  coarse  grid  problem  must  be  solved  which  is  then  inter¬ 
polated  onto  the  (fine)  solution  mesh  and  added  to  the  subdomain  solutions.  The  coarse 
grid  solve  is  critical  to  performance  of  the  method;  in  a  standard  test  case  it  was  found  to 
reduce  the  overall  iteration  count  by  a  factor  of  eight.  In  parallel,  the  coarse  grid  solve  is 
potentially  expensive  since  it  requires  communication  of  data  amongst  all  of  the  processors. 
We  have  developed  a  solver  which  requires  the  minimum  possible  number  of  message  cycles 

(log 2P  on  P  processors)  involving  messages  of  length  of  only  0{ri2)  for  an  n-point  mesh 

2 

in  two-dimensions  and  only  0(n 3)  in  three-dimensions.  This  is  a  significant  reduction  over 
previous  algorithms  in  the  literature  which  typically  have  0(n)  communication  complexity. 
Implementations  on  the  512  node  Intel  Paragon  at  Caltech  have  proven  the  viability  of  this 
approach  for  large-scale  parallel  applications. 
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